dor agent-browser: live in-Dormouse browser surface#140
Merged
Conversation
The webview CSP was `default-src 'none'` with no frame-src, so every <iframe> the IframePanel mounted was blocked outright — producing a blank white pane for any `dor iframe <url>`. Add `frame-src http: https:` (parseIframeUrl already constrains inputs to http/https). Also fix two things that sent debugging down the wrong path: - iframe command help examples used https://localhost:5173, wrong for a typical dev server; switch to http:// (+ snapshot). - IframePanel now shows a stall hint if the frame hasn't loaded within 5s, since cross-origin frames never report errors to the embedder. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The iframe surface conflicts with Dormouse's single-document input/focus model in ways mostly inherent to cross-origin frames (focused frame owns the keyboard; parent can't intercept; framing refusals give no error signal). Record these in the spec, mark the surface provisional, and note the likely path is an agent-browser surface instead of shipping iframe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add docs/specs/dor-agent-browser.md: dor ab delegates 100% to the user's agent-browser install, rendered in a Dormouse-native canvas surface (viewer client, not a fork) — so it keeps Dormouse's focus model where the iframe surface can't. Covers the workspace-scoped --key model, the session:surface registry, in-surface tab presentation, lifecycle, and the frames/input/tabs delegated channels. Cross-link it from dor-cli.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spike outcome: agent-browser's stream WS natively accepts input_mouse/ input_keyboard (verified 0.27.0), so the host CDP proxy is eliminated. Session names cannot contain '/', so managed keys map to dormouse.<workspaceId>.<key>. The stream WS rejects non-localhost Origins, so VS Code will need a loopback origin-stripping relay (Phase 7). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…(Phase 2) The session→surface registry is derived from panel/door params (single source of truth, survives webview reloads). createIframeSurface is generalized to createContentSurface, shared by iframe and agent-browser. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bs (Phases 3-7) - Frames: stream WS → createImageBitmap → canvas, with out-of-order drop and capped-backoff reconnect; placeholder states for no-session / no-page / session-ended. - Input: input_mouse/input_keyboard on the same socket, terminal-style focus semantics (forward only while selected + passthrough; the wall's dual-tap leader still escapes). - Tabs: strip fed by pushed tabs messages; switch/close via the host's allowlisted agentBrowserCommand; web-opened tabs get focused; header title follows the active tab. - Lifecycle: killing the surface closes the session (killPaneImmediately). - VS Code: CSP connect-src ws://, plus a loopback origin-stripping TCP relay in the extension host because the stream server 403s vscode-webview:// origins (verified live: 101 + frames through relay). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ution
- Mouse offset: screencast frames can be SHORTER than the viewport
(observed 1280×577 vs 1280×720 device), so scaling y by deviceHeight
stretched clicks downward. Map into frame pixels and apply one uniform
deviceWidth/frameWidth scale instead.
- Keyboard dead: clicking the non-focusable canvas blurred focus to
<body> (tabIndex=-1 ancestors don't receive click focus, and the first
click predates the interactive flip). Focus the pane explicitly on
mousedown (rAF-deferred past dockview's focus churn) and add a
window-capture fallback that forwards keys whenever the pane is
interactive but focus fell through.
- Tab actions inert: the GUI extension host's PATH lacks nvm shims, so
spawn('agent-browser') ENOENTed silently. dor ab now resolves the
absolute binary path in the invoking terminal and threads it through
the surface request → panel params → host, which falls back through
DORMOUSE_AGENT_BROWSER_BIN and PATH on ENOENT. Failures now
console.warn instead of being swallowed.
Daemon-side behavior verified live: input coords dispatch 1:1 in
viewport CSS px; tab t<N>/close work and the stream re-broadcasts tabs
on activation changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… bridge
Three daemon protocol discoveries (all verified live against 0.27.0):
- input_keyboard events WITHOUT a text field are silently dropped — which
was eating arrows, Escape, modifier keys, and every chord. Always send
text ('' for non-text keys).
- windowsVirtualKeyCode from key.charCodeAt(0) is wrong: '.' is 46 =
VK_DELETE, so periods acted as Delete (bug inherited from
agent-browser's own viewer). Proper VK map by physical key code.
- macOS native editing chords (cmd-A/C/X/Z) don't act via CDP
dispatchKeyEvent without the commands hint, which the stream protocol
drops — upstream limitation; pages still receive the chords for their
own JS shortcuts. Verified shift+Home / shift+arrows select correctly.
text is suppressed while ctrl/cmd is held so chords act instead of
inserting characters. cmd/ctrl-V now types the LOCAL clipboard into the
page (per-char keyDown replay) — plain forwarding would paste the
embedded Chromium's empty clipboard — and the Wall's pty-paste handler
yields all clipboard chords on non-terminal surfaces so it stops
swallowing cmd-V before the panel sees it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
macOS native editing chords (Cmd+C/X/A/Z) are no-ops over the stream because input_keyboard has no field mapping to CDP Input.dispatchKeyEvent.commands. Includes a verified standalone repro against agent-browser 0.27.0. To be filed at vercel-labs/agent-browser. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stream input_keyboard path can't trigger native cmd-A/C/X on macOS
(CDP drops the commands field). Add a narrow, purpose-built host
capability agentBrowserEdit(session, op) where op is selectAll|copy|cut:
the webview only picks an op name, the host owns the fixed eval JS and
runs it over the daemon's CDP connection, then writes copy/cut text to
the OS clipboard (vscode.env.clipboard). This is option (b) — no
arbitrary-eval channel exposed to the webview.
- selectAll: el.select() / execCommand('selectAll')
- copy: read selection -> OS clipboard
- cut: copy + splice-delete selection
- undo/redo deliberately left as no-ops (execCommand('undo') is
unreliable for CDP-typed input); pending the upstream commands fix
- absent on hosts that can't run the binary (standalone) -> chords fall
through to plain key forwarding so page JS shortcuts still fire
All three host-owned eval scripts verified verbatim against a live
0.27.0 session (selectAll selects, copy/cut return the right text, cut
splices the field, collapsed selection copies nothing). lib 545/545.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…el keyboard caveats Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the header screen indicator and its viewport modal, a GUI front-end for native `agent-browser set viewport` / `set device`, plus the one Dormouse-side concept (Sync-to-pane). Built leaf-up per DOR_AGENT_BROWSER.md Phase 8. - Allowlist `set` in both host command allowlists (lib + vscode-ext). - Per-surface bridge (`agent-browser-screen.ts`) connecting the pane body (state/actions) to its tab header and the modal — a surface-id controller registry + modal-open store, both useSyncExternalStore. - Panel: derive SYNCED/SCALED, ResizeObserver sync-to-pane (debounced), last-writer-wins disengage, auto-engage on create, re-sync on session restart; persist `syncEngaged` via dockview panel params (layout blob). - Header chip gated strictly on controller presence; opens the modal. - Modal (Sync / Device-registry / Custom) pre-filled from the live snapshot; suppresses Wall keys; Apply disabled with a note when the host can't run commands (Tauri). Build-forced corrections (verified against agent-browser 0.27.0): - The screencast is CSS-pixel resolution; the JPEG never encodes the browser DPR (`set viewport 800 600 2` == `@1`), so DPR is unrecoverable from frames. SYNCED is a pure viewport-CSS-size match; DPR is issued, not compared. The prior DPR-inference would have read SCALED forever on retina. - engageSync must force a re-issue even when already engaged. - Disengage only after a frame confirms the issued size landed, so a stale pre-resize frame isn't mistaken for an external override. - Call host adapter methods through the instance, not a detached reference — `const cmd = getPlatform().agentBrowserCommand; cmd(...)` dropped `this` and threw on `this.requestResponse`, silently breaking every webview→host command (set, tab switch/close, edit chords). Fixed at both call sites. Spec + plan updated for the persistence channel and DPR findings. New unit tests for the screen registry; lib suite green (549). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… pulses The stream's screencast is CSS-resolution only — Chromium's Page.startScreencast captures in DIP and offers no deviceScaleFactor/scale knob (verified against the CDP spec, crbug 781117, and probes: our own screencast at DSF=2 still returns 1x; only Page.captureScreenshot honors DPR). So on Retina the screencast upscales to mush regardless of who drives it. agent-browser's own `screenshot` command, by contrast, is device-resolution (2560x1600 for a 1280x800@2 pane) and ~17fps via the daemon's existing connection — no second CDP client, no session→Chrome mapping, stays a viewer. So: display device-resolution screenshots, and use screencast frames purely as "page changed" pulses signalling when to grab a fresh one. Frames are no longer decoded/drawn (just read for viewport + indicator), which also drops per-frame decode work. Backpressure (latest-only, self-throttling): - at most one screenshot in flight; a pulse during a shot sets `dirty` (no queue) so bursts collapse to one follow-up, latest wins; - a seq guard drops any out-of-order decode so a stale shot never paints over a newer one; - next shot waits ~1.5x the measured (EWMA) capture time since the last start, so slow capture backs off (~2/3 duty), with a 50ms floor against tight loops; - a static page produces no pulses, hence no shots and no cost. Host capability `agentBrowserScreenshot` (vscode-ext): runs `screenshot` to a reused temp file and returns the bytes base64. Falls back to rendering the screencast frames where the host can't screenshot (Tauri). Also bind the agent-browser adapter methods in the constructor — detached `getPlatform().agentBrowserX` references drop `this` and throw on requestResponse; this had to be fixed at three call sites already. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spec (docs/specs/dor-agent-browser.md): - New "Headed Pop-Out" section. Pop-out relaunches a surface's browser headed as a real OS window the user drives directly; the pane becomes a clean placeholder (no screencast of the headed window — sidesteps headed-screencast / off-screen-occlusion / window-tracking). GUI-only, randomKillChar()- confirmed. Identity-preserving relaunch keeps the session name so `dor ab` keeps targeting it. v1 carries ordered tab URLs only (profile/login persistence is a named follow-up spike). Best-effort positioning over the pane rect, center-on-monitor fallback (always in VS Code, always on Wayland). No control tab: the headed window ending by any gesture auto-reverts to a headless surface reopening the last non-empty tab list; only `dor kill` tears down. Platform-gated capability matrix (Tauri-first; VS Code can't position; Wayland can't self-position; web hides it). - Corrected two now-stale parts to match shipped code: "Frames (out)" now documents the screenshot-pulse pipeline (was screencast-to-canvas), and the Host capabilities list gains agentBrowserScreenshot/agentBrowserEdit + a planned pop-out entry. Plan (DOR_AGENT_BROWSER.md): - Phase 9 (DONE) records the HiDPI screenshot streaming as shipped. - Phase 10 sequences headed pop-out leaf-up (host capability, affordance+confirm, relaunch+reopen+position, placeholder, lifecycle, cross-platform), plus a profile-persistence spike. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # dor/src/node-runtime.d.ts # lib/src/components/Wall.tsx
The spec was accreting planned work inline. Restructure it into a concise description of what's actually built (Delegation, --key, mapping, tabs, screen indicator, lifecycle, channels, touchpoints, host capabilities, CSP), with a clearly-separated "Future Expansions" section at the bottom holding the Headed Pop-Out design and other planned items (profile persistence, CLI sync verb, undo/redo). Tightened the completed-feature prose along the way. Slim DOR_AGENT_BROWSER.md to match: the built phases now live in the spec, so the plan doc drops the completed-phase log and keeps only the forward build sequence (Phase 10 pop-out + profile spike) plus the dev loop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
We display device-resolution screenshots and use stream frames only as "page changed" pulses, so a frame's base64-JPEG payload is thrown away. On an animating page that's ~13 MB/s of frame data at 1080p/60fps being JSON.parsed (and allocated as a throwaway ~216 KB string) on the webview main thread, where it competes with screenshot rendering. Probed the screencast: rate is ~60fps independent of viewport size, and there's no resolution/fps/quality knob (only AGENT_BROWSER_STREAM_PORT) — frame size is the viewport, which our screenshots share, so we can't shrink it. But we don't have to parse it: frames are the only large messages, so pulse on any message over a 16 KB threshold and skip the parse + allocation entirely. The viewport (for the indicator + input mapping) already comes from the small `status` messages, which fire on every change. Smaller messages still parse, so a rare tiny-viewport frame under the cutoff still pulses correctly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Transport: ship screenshot frames as raw bytes (Uint8Array over the webview
structured-clone channel) instead of base64. Removes the host-side
toString('base64'), the webview atob + per-byte decode loop, and ~33% transfer
inflation per shot — the largest, most frequent payload in the surface.
Dogfood-confirmed VS Code preserves the Uint8Array across the boundary.
Quality (pure extractions, behavior-preserving):
- use-pane-chrome.ts: the pane register/unregister + spawn-animation effects,
previously copy-pasted byte-for-byte in TerminalPanel/IframePanel/
AgentBrowserPanel, are now one shared hook.
- agent-browser-input.ts: the pure keyboard/mouse→CDP translation (VK tables,
virtualKeyCode, modifiers, mouse maps, edit ops) lifted out of the panel, with
agent-browser-input.test.ts covering the '.'=VK_DELETE footgun + modifier mask.
- agent-browser-screenshot-loop.ts: the backpressure state machine as
createScreenshotLoop({getSession,getBinaryPath,isCapable,draw}); the panel
wires it in a one-line useMemo.
AgentBrowserPanel drops ~190 lines and now reads as transport + sync + render.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Storybook story for the screen/viewport modal, backed by a standalone mock ScreenController (fixed, memoised snapshot — no registry). Variants cover the pre-selection branches and degradation: Synced (pre-selects Sync), ScaledCustom (pre-selects Custom prefilled with live dims), PhoneViewport, and HostIncapable (Apply disabled + dor-ab note). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…=scaled) Replace the SYNCED/SCALED text chip with header-sized icons (FrameCornersIcon for SYNCED, ResizeIcon for SCALED), matching the other header action buttons. State stays conveyed via aria-label + tooltip. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phases 1–9 are landed and the forward-looking pop-out design lives in the spec (docs/specs/dor-agent-browser.md → Headed Pop-Out), so the working plan doc has served its purpose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The loopback relay previously accepted any /stream/<port> upgrade, so any
local process could pipe to an arbitrary stream port through it. Mint a
short-lived, single-use token+port grant per surface: getAgentBrowserStreamUrl
now hands back ws://127.0.0.1:<relayPort>/stream/<streamPort>/<token>, and the
relay rejects (403) any request whose token/port doesn't match an unexpired
grant, consuming it on use. Grants are swept lazily.
Also:
- reset knownTabIds on (re)connect so stale tab ids don't suppress popups
after a stream reconnect
- dedup AgentBrowser{Command,Edit,Screenshot}Result / EditOp and the
subcommand allowlist in agent-browser-host against the canonical
definitions in lib platform/types (single source of truth)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploying mouseterm with
|
| Latest commit: |
72de860
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://23c46cb5.mouseterm.pages.dev |
| Branch Preview URL: | https://cli-expanded.mouseterm.pages.dev |
dormouse-bot
left a comment
dormouse-bot
left a comment
Collaborator
There was a problem hiding this comment.
Read the full change — CLI passthrough, the canvas viewer, the screenshot/screencast pacing, and the tokenized loopback relay all hang together cleanly, and the CLI + input/screen unit tests cover the tricky parts (the .=VK_DELETE footgun, key/session resolution, surface reuse). One genuinely nice touch: the relay strips Origin, dials only an explicitly-granted 127.0.0.1:<port>, and gates on a single-use 256-bit token with a TTL — no notes there.
Only nits below: two doc comments still say the screenshot is returned "base64-encoded", but the transport is now raw Uint8Array bytes over structured clone (the PR's own "no base64 round-trip"). Worth correcting so a future reader isn't misled. Non-blocking.
Future Expansions proposal for the agent-browser surface header: mirror Chrome's toolbar (sync chip far-left, then back/forward/refresh, then URL), show the URL instead of the HTML title, badge non-default --key surfaces, and — the differentiated piece — surface the dev-server connection by correlating a loopback :port to the Dormouse terminal pane serving it (jump-to-terminal). Verified against agent-browser 0.27.0 and the codebase: - back/forward/reload are native commands (history nav confirmed) → just allowlist them on agentBrowserCommand, no eval fallback. - getOpenPorts is fully plumbed + tested but has zero UI consumers — the per-OS port detection is done; the correlation store/chip are greenfield. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…review)
The screenshot transport is raw Uint8Array bytes over structured clone, not
base64. Correct the two flagged doc comments (platform types, agent-browser-host)
plus the same stale wording in the spec's host-capabilities list, and tighten the
CSP note ("in-memory image bytes via createImageBitmap" rather than "base64").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dormouse-bot
approved these changes
Jun 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
dor agent-browser(aliasdor ab) — a live, interactive browser rendered inside Dormouse by delegating 100% to the user's own agent-browser install. It's a viewer client, not a fork: Chromium, CDP, the screencast, and the entire command surface stay in agent-browser; Dormouse adds a thin surface that renders the session to a<canvas>, forwards input, and presents tabs.This is the chosen alternative to the iframe surface — because the browser renders to a Dormouse-owned
<canvas>rather than a cross-origin<iframe>, Dormouse keeps its own keydown listener and never loses focus control.Full design:
docs/specs/dor-agent-browser.md.How it fits together
dor ab <args...>is a near-transparent passthrough toagent-browser <args...>, intercepting only--key(workspace-scoped browser identity, defaultdefault) /--session(raw escape hatch). After a successful command it reads the stream port and opens/reuses the bound surface (one session ⇄ one surface).agent-browserjoinsterminal/iframeas a first-class surface type viacreateContentSurface; shared pane chrome extracted tousePaneChrome.input_mouse/input_keyboardin. Frames are used as "page changed" pulses; the panel displays crisp device-resolution screenshots (the screencast is CSS-resolution only). Includes a SYNCED/SCALED screen indicator + viewport modal.agentBrowserCommand(allowlisted subcommands),agentBrowserScreenshot(binary transport, no base64 round-trip),agentBrowserEdit(native select-all/copy/cut chords CDP drops over the stream), and a loopback stream relay that strips the rejectedvscode-webview://origin. The relay is gated by short-lived, single-use token+port grants.🤖 Generated with Claude Code